Repository navigation
Conversation
… holdouts Registers every page of the sealed holdouts (DocLayNet, PubTables-1M and FinTabNet test splits, ParseBench, DP-Bench, olmOCR-bench) by file hash, normalised text hash, render hash and source document id. Hashes only; the olmOCR-bench pages were hashed by program and never read. `python -m bench.registry check` fails when a holdout page, or a page of a holdout document, appears in a dev, validation or training manifest, and names the page hash, the holdout and the layer. A source document may not span the train side (dev, training) and validation. A dataset that sits in a layer directory without being registered in layers.json fails the check. A holdout that is missing or only partly fingerprinted prints `SKIPPED (not a pass)` and exits 3; CI turns that into a warning annotation. Planted-leak fixtures (renamed copy, re-render at two DPIs, dirty scan, other PDF container) and mutation tests for each match rule run in the pipeline. PubTables-1M pages are registered by byte hash and document id only: they are cut with mutool 1.23.10 and are not materialised here. Refs pdfluent/pdfluent-internal#971 Signed-off-by: jasperdew <10383561+jasperdew@users.noreply.github.com>
…lter Harvests arXiv metadata through the OAI-PMH endpoint (arXivRaw), at most one request every three seconds on a single connection, resumable per window, with the project and a contact address in the User-Agent. The requester-pays S3 bucket is not used. A paper is kept only when its metadata licence is exactly one of the CC0 1.0, CC BY or CC BY-SA URLs; every other value is rejected and counted in a reject report. CC BY-SA papers carry share_alike. Papers whose arXiv id appears in a sealed holdout of the page-hash registry (olmOCR-bench among them) are rejected. The sample draw alternates math-heavy and other categories and round-robins over primary categories, so any prefix is spread. PDF and HTML download is gated: arXiv's OAI data policy does not permit full-content harvesting without an arrangement, and robots.txt asks for a 15 s crawl delay and disallows /e-print. `fetch` refuses without ARXIV_CONTENT_ARRANGEMENT, reads robots.txt, never requests a disallowed path and resumes from a ledger. The terms as read are in datasets/arxiv/EVIDENCE.md. Fixtures: one paper per licence value (exactly CC0, CC BY, CC BY-SA pass), a planted olmOCR-bench id, and mutations of the allow-list, all in the pipeline. Refs pdfluent/pdfluent-internal#972 Signed-off-by: jasperdew <10383561+jasperdew@users.noreply.github.com>
…papers Metadata of all 3,224,546 records harvested through OAI-PMH. 665,925 papers pass the CC0 / CC BY / CC BY-SA filter after the olmOCR-bench exclusion (31,396 of them CC BY-SA, 165,777 math-heavy by primary category, 382,574 with an id from 2401 on). The reject report lists the count per licence value. The sample has 2,000 papers from 2401 on, half math-heavy, spread over 149 primary categories, with licence URL, authors, title and version per paper. PDF, HTML and source have not been fetched: the content route is gated, see datasets/arxiv/EVIDENCE.md. The sample manifest is exempt from the tool-name rule: an author surname and a paper title match it. Refs pdfluent/pdfluent-internal#972 Signed-off-by: jasperdew <10383561+jasperdew@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3 (feat/odl-bench-job-960-v2). Do not merge before #3.
#971 page-hash registry and leak check
bench/registry.py,datasets/registry/(hashes only): every page of DocLayNet, PubTables-1M and FinTabNet test splits, ParseBench, DP-Bench and olmOCR-bench by file hash, normalised text hash, render hash and source document id. The 19-axes holdout is thehalf: holdoutpages of the first four.python -m bench.registry checkfails when a holdout page, or a page of a holdout document, is in dev, validation or training, and names the page hash, the holdout and the layer. A document may not span train (dev, training) and validation. An unregistered dataset indatasets/layers/fails.SKIPPED (not a pass)per holdout and exits 3; CI turns that into a warning annotation, pytest into a skip line. Currently incomplete: docling-dpbench (not on disk), PubTables-1M (mutool 1.23.10 seal, registered by byte hash and document id only), ParseBench (1800 of 2039 pages not on disk), DocLayNet (51 pages without text or render).#972 arXiv harvest
bench/arxiv.py: OAI-PMHarXivRaw, one request per 3 s, one connection, resumable. Licence allow-list is the exact CC0 / CC BY / CC BY-SA URLs; olmOCR-bench ids are excluded through the registry.datasets/arxiv/EVIDENCE.md.CI
tests/test_round.py, 2 intests/test_workflows.pybecausecommit-identity.ymlruns onpull_requeston a hosted runner;fix/fork-runner-testaddresses the latter). Not touched here.engine/corpus/structured/FINTABNET.jsonis untracked on the base checkout; the registry was built from that copy.